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Report  No.  96-037  December  11,  1995 

MEMORANDUM  FOR  ASSISTANT  SECRETARY  OF  THE  NAVY  (FINANCIAL 

MANAGEMENT  AND  COMPTROLLER) 
COMMANDER,  MARINE  CORPS  COMBAT 
DEVELOPMENT  COMMAND 
COMMANDER,  MARINE  CORPS  SYSTEMS  COMMAND 

SUBJECT:  Audit  of  Cost  and  Operational  Effectiveness  Analysis  for  Marine  Corps 
Third  Echelon  Test  Set  (Project  No.  5LB-5039) 


Introduction 

We  are  providing  this  report  for  information  and  use.  This  audit  was  performed 
in  response  to  a  congressional  request  from  Senator  Howell  Heflin  to  review  the 
accuracy  of  the  Marine  Corps  cost  and  operational  effectiveness  analysis 
(COEA)  for  a  third  echelon  test  set  (TETS).  The  Senator  alleged  that  the 
COEA  showed  a  decided  bias  toward  the  Marine  Corps'  proposed  system 
alternative,  referred  to  in  the  COEA  as  the  automated  test  support  unit 
prototype.  The  Senator  expressed  concern  that  the  Marine  Corps  intended  to 
issue  a  request  for  proposal  for  a  TETS  based  on  the  findings  of  the  COEA. 
The  TETS  acquisition  program  is  an  acquisition  category  IV  program,  and  has 
an  estimated  total  life  cycle  cost  of  nearly  $208  million.  (See  glossary  in 
Enclosure  1.) 


Audit  Results 

The  allegation  was  not  substantiated.  The  Marine  Corps  used  a  COEA  to 
support  the  acquisition  of  a  TETS  that  was  adequate  in  its  methodology, 
objective  in  its  evaluation  of  system  alternatives,  and  reliable  in  its  conclusions 
and  recommendations.  The  Marine  Corps'  acquisition  decision  process 
effectively  evaluated  automatic  test  system  alternatives  for  satisfying  TETS 
requirements.  Management  controls  applicable  to  the  performance  and  review 
of  the  COEA  process  were  adequate  in  that  we  identified  no  material 
management  control  weaknesses. 


Audit  Objectives 

The  overall  audit  objective  was  to  review  the  accuracy  of  the  COEA  used  by  the 
Marine  Corps  to  justify  the  purchase  of  an  automatic  test  system,  specifically  a 
TETS.  Special  emphasis  was  placed  on  validating  the  Marine  Corps  acquisition 
decision  process  that  resulted  in  the  decision  to  acquire  an  automatic  test  system 
based  on  the  automated  test  support  unit  prototype  over  other  alternatives.  We 
also  reviewed  the  management  control  program  as  it  related  to  the  audit 
objectives. 
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Scope  and  Methodology 

Scope  and  Methodology.  We  reviewed  data  from  December  1991  through 
September  1995  to  accomplish  our  audit  objectives.  We  reviewed  the 
methodology  used  to  develop  the  TETS  COEA,  including  the  COEA  study 
directive  and  analysis  plan.  We  also  interviewed  the  COEA  study  review  team. 
We  analyzed  the  COEA  for  overall  accuracy  and  compliance  with  established 
DoD  and  Department  of  the  Navy  policies  and  procedures  for  preparing 
COEAs.  We  interviewed  DoD,  Army,  Navy,  Marine  Corps,  and  contractor 
officials  concerning  the  TETS  program  and  automatic  test  system  policy. 

We  reviewed  the  Marine  Corps'  decision  process  for  evaluating  automatic  test 
system  alternatives  to  satisfy  third  echelon  testing  requirements.  We  reviewed 
data  the  Marine  Corps  considered  in  deciding  to  pursue  a  TETS  solution.  We 
reviewed  the  capabilities  of  alternative  systems  analyzed  in  the  COEA  and 
witnessed  demonstrations  of  the  automated  test  support  unit  prototype,  the  direct 
support  electrical  system  test  set,  general  purpose  interface  assembly  (direct 
support  test  set),  and  the  Army  integrated  family  of  test  equipment  (IFTE)  base 
shop  test  facility.  We  evaluated  the  conclusions  of  the  test  measurement  and 
diagnostic  study  on  automatic  test  system  for  Marine  Corps  ground  weapon 
systems,  the  results  of  the  automated  test  support  unit  prototype  concept 
demonstration,  the  conclusions  and  recommendations  of  the  COEA,  the  findings 
of  the  Institute  for  Defense  Analyses  study  on  DoD  investment  strategy  for 
automatic  test  system,  the  conclusions  of  the  concept  studies  performed  on  the 
Navy  consolidated  automated  support  system  (CASS)  and  the  Army  IFTE,  and 
DoD  automatic  test  system  policy  and  the  significance  each  had  on  the  Marine 
Corps  acquisition  strategy. 

Audit  Period,  Standards,  and  Locations.  This  economy  and  efficiency  audit 
was  performed  from  May  to  September  1995.  We  conducted  this  audit  in 
accordance  with  auditing  standards  issued  by  the  Comptroller  General  of  the 
United  States  as  implemented  by  the  Inspector  General,  DoD.  We  included 
tests  of  management  controls  considered  necessary.  We  did  not  use 
computer-processed  data  or  statistical  sampling  procedures  for  this  audit. 
Enclosure  6  lists  the  organizations  visited  or  contacted. 


Management  Control  Program 

DoD  Directive  5010.38,  "Internal  Management  Control  Program," 
April  14,  1987,  requires  DoD  managers  to  implement  a  comprehensive  system 
of  management  controls  that  provides  reasonable  assurance  that  programs  are 
operating  as  intended  and  to  evaluate  the  adequacy  of  those  controls. 

Scope  of  Review  of  Management  Control  Program.  We  reviewed 
management  control  procedures  regarding  the  performance  and  review  of  the 
COEA  process.  We  did  not  assess  the  adequacy  of  management's 
self-evaluation  of  those  controls  because  no  discrepancies  were  noted. 
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Adequacy  of  Management  Controls.  Management  controls  associated  with 
the  performance  and  review  of  the  COEA  process  were  adequate  as  they  applied 
to  the  audit  objectives. 


Prior  Audits  and  Other  Reviews 

The  Inspector  General,  DoD,  has  issued  five  prior  audit  reports  related  to 
automatic  test  systems  within  the  last  5  years.  In  addition,  we  considered  two 
other  reviews  concerning  automatic  test  systems.  See  Enclosure  5  for  a 
summary  of  the  reports  and  reviews. 


Background 

TETS  Operational  Concept.  The  TETS  operational  concept  is  to  provide  a 
portable  test  capability  for  the  Marine  Corps  at  or  near  the  forward  edge  of  the 
battlefield  area.  Enclosure  2  depicts  the  TETS  operational  concept  and 
describes  the  Marine  Corp's  third  echelon  level  of  operational  maintenance. 
The  TETS  operational  requirements  document  specifies  that  TETS  will  have  the 
capability  to  test,  diagnose,  and  screen  electronic  equipment  for  a  wide  variety 
of  weapon  systems.  The  TETS  must  be  relatively  compact,  lightweight,  and 
rugged  and  constructed  from  commercial-off-the-shelf,  modified  commercial- 
off-the-shelf,  or  nondevelopmental  item  equipment.  Enclosure  3  is  a  brief 
history  of  the  TETS  program. 

COEA.  COEAs  are  prepared  to  aid  decisionmakers  by  highlighting  the  relative 
advantages  and  disadvantages  of  the  alternatives  being  considered.  The  TETS 
COEA  was  prepared  in  January  1994  to  support  the  milestone  I,  concept 
demonstration,  approval  for  TETS  .  The  TETS  COEA  was  conducted  to  aid 
Marine  Corps  decisionmakers  in  judging  whether  any  of  the  proposed 
alternatives  to  the  current  program  offer  sufficient  military  benefit  to  be  worth 
the  cost.  Enclosure  4  is  a  discussion  of  efforts  being  taken  to  improve  the 
COEA  process. 

The  TETS  COEA  considered  the  following  alternatives: 
o  an  automated  test  support  unit  prototype; 
o  a  base  case; 
o  the  CASS; 

o  the  direct  support  test  set; 
o  an  IFTE  base  shop  test  facility. 


xIn  May  1995,  the  milestone  I  (concept  demonstration)  approval  was  combined 
with  milestone  II  (development)  approval. 
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The  TETS  COEA  recommended  that  Marine  Corps  decisionmakers: 

o  Eliminate  CASS  and  IFTE  base  shop  test  facility  because  of  the 
dominance  of  the  automated  test  support  unit  prototype  and  the  direct  support 
test  set  in  the  areas  of  cost  and  capability. 

o  Select  between  the  automated  test  support  unit  prototype  and  the  direct 
support  test  set  based  on  trade-offs  between  availability,  capability,  cost,  and 
uncertainty. 

Criteria.  The  criteria  for  developing  COEAs  are  contained  in  DoD  and 
Secretary  of  the  Navy  guidance.  DoD  Components  are  to  implement  the 
following  criteria  when  developing  COEAs. 

o  DoD  Instruction  5000.2-M,  part  4,  section  E,  "Cost  and  Operational 
Effectiveness  Analysis,"  February  23,  1991,  establishes  policies  and  procedures 
that  form  the  basis  for  developing  COEAs  to  support  milestone  decision 
reviews. 

o  DoD  Manual  5000. 2M,  part  8,  "Cost  and  Operational  Effectiveness 
Analysis,"  February  23,  1991,  provides  general  procedures  and  guidelines  for 
developing  COEAs. 

o  Secretary  of  the  Navy  Instruction  5000. 2A,  part  8,  "Cost  and 
Operational  Effectiveness  Analysis,"  December  9,  1992,  establishes  COEA 
responsibilities  and  procedures  for  the  Department  of  the  Navy. 

Automatic  Test  System  Policy.  In  April  1994,  the  Deputy  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  issued  a  policy  memorandum  on 
automatic  test  systems.  The  policy  requires  DoD  Components  to  satisfy  all 
acquisition  needs  for  automatic  test  systems  by  using  designated  automatic  test 
system  families  or  to  obtain  a  waiver.  It  was  established  to  minimize  unique 
types  of  automatic  test  system  being  introduced  into  DoD  field,  depot,  and 
manufacturing  operations,  and  to  encourage  use  of  commercial-off-the-shelf 
testers  and  components,  thereby  reducing  the  logistics  burdens  and  long-term 
costs  to  DoD.  The  Navy  CASS  and  the  Army  IFTE  were  designated  as  the 
initial  DoD  families.  Enclosure  4  further  discusses  issues  concerning  automatic 
test  system  policy. 


Discussion 

Overall  Accuracy  of  the  TETS  COEA.  The  results  of  the  TETS  COEA 
showed  that  it  was  adequate  in  its  methodology,  objective  in  its  evaluation  of 
system  alternatives,  and  reliable  in  its  conclusions  and  recommendations. 

COEA  Methodology.  The  Marine  Corps  used  a  methodology  consistent 
with  established  DoD  and  Service  guidance  for  developing  COEAs.  The 
Marine  Corps  independently  performed  and  appropriately  tailored  the  COEA  for 
the  program  milestone  review  being  supported,  specifically,  the  milestone  I, 
concept  demonstration,  approval.  With  the  COEA,  the  Marine  Corps  identified 
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the  mission  need,  key  constraints,  and  the  automatic  test  system  alternatives.  It 
evaluated  the  alternatives  in  the  areas  of  availability,  capability,  cost,  and 
uncertainty. 

Evaluation  of  System  Alternatives.  Using  the  COEA,  the  Marine 
Corps  evaluated  the  system  alternatives  objectively  and  consistently  based  on 
available  information.  The  study  analysis  team  collected  data  from  system 
documentation  and  recorded  performance  and  personal  observation  of  system 
demonstrations.  It  evaluated  the  system  alternatives  using  approved  measures  of 
effectiveness  based  on  the  TETS  requirements.  The  results  of  the  COEA 
highlighted  the  advantages  and  disadvantages  of  each  alternative,  and  presented 
reliable  conclusions  and  recommendations  on  the  proposed  alternatives  for  use 
by  acquisition  decisionmakers. 

COEA  Conclusions  and  Recommendations.  The  conclusions  and 
recommendations  derived  from  the  COEA  provided  reliable  information  to 
acquisition  decisionmakers.  The  conclusions  and  recommendations  were 
adequately  supported  by  availability,  capability,  cost,  and  uncertainty  analyses. 
The  analyses  clearly  identified  the  automated  test  support  unit  prototype  and  the 
direct  support  test  set  as  the  dominant  system  alternatives  that  could  satisfy  the 
requirements  of  TETS.  The  resulting  conclusions  of  the  COEA  were 
substantiated  by  concept  studies  on  CASS  and  IFTE  conducted  in  March  1995, 
as  part  of  a  waiver  review  for  TETS. 

Marine  Corps  Acquisition  Decision  Process.  The  Marine  Corps  acquisition 
decision  process  effectively  evaluated  automatic  test  system  alternatives  for 
satisfying  TETS  requirements.  The  Marine  Corps  selection  of  the  automated 
test  support  unit  prototype  over  other  alternative  solutions  was  supported  by  a 
number  of  factors.  The  most  significant  of  the  factors  are  discussed  in  the 
following  paragraphs. 

TETS  Concept  Demonstration.  The  TETS  concept  demonstration  was 
a  significant  factor  in  the  Marine  Corps  decision  to  pursue  a  TETS  acquisition. 
In  September  1992,  the  automated  test  support  unit  prototype  demonstrated  that 
a  system  constructed  from  commercial-off-the-shelf  equipment  could  fulfill  the 
TETS  requirements.  The  results  of  the  TETS  concept  demonstration  prompted 
the  Commander,  Marine  Corps  Systems  Command,  to  establish  TETS  as  a 
program. 

TETS  COEA.  The  Marine  Corps  conducted  a  COEA  in  1993,  to 
determine  a  cost-effective  solution  to  support  all  Marine  Corps  programs  that 
require  automatic  test  equipment.  The  COEA  provided  Marine  Corps 
decisionmakers  with  information  on  the  advantages  and  disadvantages  of 
alternative  systems  for  satisfying  automatic  testing  requirements.  Based  on  the 
results  of  the  COEA  and  trade-off  analyses  between  the  automated  test  support 
unit  prototype  and  the  direct  support  test  set,  Marine  Corps  decisionmakers 
determined  that  the  automated  test  support  unit  prototype  solution  provided  the 
most  affordable,  operationally  effective,  and  reliable  approach. 
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Compliance  With  DoD  Automatic  Test  System  Policy.  DoD 
automatic  test  system  policy,  issued  on  April  29,  1995,  significantly  impacted 
the  TETS  acquisition  decision.  As  stated  earlier,  DoD  policy  for  automatic  test 
system  required  that  DoD  Components  satisfy  all  acquisition  needs  for  an 
automatic  test  system  by  using  designated  CASS  and  IFTE  families,  unless  a 
waiver  was  granted. 

The  Marine  Corps  presented  the  TETS  program  to  the  automatic  test 
system  management  board  in  the  fall  of  1994,  for  waiver  recommendation.  The 
management  board  reviewed  the  automated  test  support  unit  prototype  and 
requested  that  concept  studies  be  conducted  on  the  CASS  and  IFTE  programs  to 
determine  the  feasibility  of  a  satisfactory  variant  to  satisfy  the  TETS 
requirements.  Both  the  CASS  and  IFTE  concept  studies  concluded  that  neither 
CASS  nor  IFTE  could  satisfy  the  power,  reconfigurability  and  portability,  size, 
and  weight  requirements  of  TETS.  On  May  1,  1995,  the  management  board 
granted  a  waiver  for  TETS,  allowing  the  Marine  Corps  Systems  Command  to 
procure  an  automatic  test  system  that  can  satisfy  the  TETS  requirements  through 
open  bid  sample  competition.2 


Conclusion 

The  TETS  COEA  was  conducted  in  accordance  with  established  DoD  and 
Department  of  the  Navy  policies  and  procedures.  The  allegation  of  bias  toward 
the  Marine  Corps'  proposed  concept  solution  for  a  TETS  was  unsubstantiated. 
The  conclusions  of  the  COEA  were  reliable  and  validated  by  subsequent  studies 
performed  independent  of  the  COEA  process.  The  Marine  Corps  decision  to 
pursue  a  TETS  acquisition  was  based  on  an  effective  review  of  alternative 
systems  and  was  compliant  with  DoD  policy  on  automatic  test  systems. 


Management  Comments 

We  provided  a  draft  of  this  report  to  you  on  October  20,  1995.  Because  this 
report  contains  no  findings  or  recommendations,  comments  were  not  required, 
and  none  were  received.  Therefore,  we  are  publishing  this  report  in  final  form. 


2Open  bid  sample  competition  is  the  process  the  Marine  Corps  will  employ  in 
selecting  a  TETS  vender.  It  includes  the  submission  of  bids  by  all  interested 
parties,  selection  of  the  most  promising  bids  (three  to  five),  and  submission  of 
actual  prototypes  for  testing  before  a  contract  is  awarded. 
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We  appreciate  the  courtesies  extended  to  the  audit  staff.  For  additional 
information  on  this  report,  please  contact  Mr.  John  A.  Gannon,  Audit  Program 
Director,  at  (703)  604-9427  (DSN  664-9427)  or  Mr.  Stephen  T.  Hampton, 
Acting  Audit  Project  Manager,  at  (703)  604-9428  (DSN  664-9428). 
Enclosure  7  lists  the  distribution  of  this  report.  The  audit  team  members  are 
listed  on  the  inside  back  cover. 

David  K.  Steensma 
Deputy  Assistant  Inspector  General 
for  Auditing 

Enclosures 
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Automated  test  support  unit  prototype.  The  automated  test  support  unit  was 
one  of  the  alternative  automatic  test  systems  considered  in  the  TETS  COEA.  It 
was  developed  to  show  the  viability  of  a  commercial-off-the-shelf  solution  for 
TETS;  and  successfully  used  in  the  TETS  concept  demonstration  held  in 
September  1992.  The  prototype  consisted  of  integrated  commercial-off-the- 
shelf  equipment  with  Government  developed  equipment  interfaces  and  test 
software.  It  used  industry  standard,  which  allowed  for  communication  among 
devices,  ease  of  programming  integration,  and  physical  size  and  software  cost 
reduction  through  the  use  of  a  common  interface.  The  automated  test  support 
unit  prototype  allowed  interfacing  of  components  manufactured  by  multiple 
vendors. 

Automatic  test  equipment.  Automatic  test  equipment  consists  of  an  operating 
system  or  executive  software,  which  runs  on  a  main  computer,  and  a  range  of 
hardware  components.  Components  of  automatic  test  equipment  hardware  are  a 
main  computer  within  the  test  equipment,  an  operating  system,  measurement 
and  stimulus  instruments,  a  signal  control  and  switches,  and  needed  interfaces 
such  as  heating  and  cooling  sources  and  structural  supports. 

Automatic  test  system.  An  automatic  test  system  is  comprised  of  automatic 
test  equipment  hardware,  operating  software,  and  test  program  sets  that  are 
needed  to  test  individual  weapon  system  electronics.  It  is  used  in  DoD  field  and 
depot  electronics  maintenance  organizations  to  test  electronics  equipment  that 
would  be  difficult  or  impossible  to  test  manually. 

Base  case.  The  base  case,  as  referred  to  in  the  TETS  COEA,  represents  the 
status  quo  and  consists  of  the  test  equipment  that  is  performing  the  Marine 
Corps  automatic  test  functions.  It  is  first  generation  electronics  technology  and 
is  old,  difficult  to  maintain,  and  it  takes  too  long  to  perform  diagnostic  tasks. 

Commercial-off-the-shelf.  Equipment  that  can  be  purchased  through 
commercial  retail  or  wholesale  distributors. 

Consolidated  automated  support  system.  CASS  is  an  alternative  system 
considered  in  the  TETS  COEA.  The  Navy's  CASS  is  general  purpose, 
standardized  automatic  test  equipment  composed  of  six  electronic 
configurations.  The  automatic  test  equipment  is  structured  around  a  common 
core  with  computer-assisted,  multifunctional  capabilities  to  support  testing  of 
aircraft  subsystems  and  missiles.  The  CASS  program  features  fleet-wide 
standardization  of  hardware  and  software  elements  that  is  designed  to  enhance 
electronic  test  capability  at  the  intermediate  and  depot  maintenance  levels. 
CASS  is  one  of  the  designated  families  under  DoD  automatic  test  system  policy. 

Direct  support  electrical  system  test  set,  general  purpose  interface 
assembly.  This  system,  developed  by  Pentastar  Electronics,  Incorporated,  was 
considered  in  the  TETS  COEA.  It  is  a  small,  rugged  automated  test  unit  that 
can  be  transported  forward  in  tactical  vehicles  to  support  units  engaged  in 
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combat  operations.  It  supports  the  Ml  Abrams  Tank  and  M2/M3  Bradley 
Fighting  Vehicles.  It  is  constructed  of  proprietary  components  and  uses  a 
Pentastar  proprietary  operating  system. 

Integrated  family  of  test  equipment  base  shop  test  facility.  The  IFTE  base 
shop  test  facility  is  an  alternative  automatic  test  system  considered  in  the  TETS 
COEA.  IFTE  is  composed  of  five  subsystems  of  automatic  test  equipment. 
The  Army  initiated  the  IFTE  to  minimize  unique  automatic  test  equipment  that 
it  was  procuring  for  weapon  systems.  The  primary  intermediate  maintenance 
level  subsystem  of  IFTE  is  the  base  shop  test  facility,  which  is  installed  in  an 
S-280  shelter  and  mounted  on  a  5-ton  truck  for  Army  use.  It  is  general 
purpose,  multifunctional  automatic  test  equipment  that  uses  several 
configurations  of  tests  to  diagnose  electronic  faults  or  failures  in  weapon 
systems.  IFTE  is  one  of  the  designated  families  under  DoD  automatic  test 
system  policy. 

Modified  commercial-off-the-shelf  equipment.  Modified  commercial-off-the- 
shelf  equipment  is  equipment  that  has  been  customized  to  ' meet  functional 
requirements. 

Nondevelopmental  item  equipment.  Nondevelopmental  item  equipment  is 
equipment  that  can  be  commercial-off-the-shelf,  ruggedized  or  militarized.  It  is 
defined  as: 

o  an  item  of  supply  that  is  available  in  the  commercial  marketplace. 

o  a  previously  developed  item  of  supply  that  is  in  use  by  a  department 
or  agency  of  the  United  States,  a  state  or  local  Government,  or  a  foreign 
Government  with  which  the  United  States  has  a  mutual  defense  cooperation 
agreement. 

o  an  item  that  requires  only  minor  modification  to  meet  the  procuring 
agency’s  requirements. 

o  an  item  being  produced  that  does  not  meet  other  nondevelopmental 
item  requirements  solely  because  it  is  not  yet  in  use,  or  not  yet  available  in  the 
commercial  market  place. 

Third  echelon  test  set.  The  TETS  is  a  portable  automatic  test  capability  to  be 
used  by  the  Marine  Corps  at  or  near  the  forward  edge  of  the  battlefield  area.  It 
will  consist  of  237  base  configurations.  Of  its  237  configurations,  165  will 
include  a  radio  frequency  test  capability  and  41  will  include  an  electro-optical 
test  capability. 
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Third  Echelon  Maintenance.  The  Marine  Corps  authorizes  specially  trained 
personnel  to  perform  third  echelon  maintenance  at  or  near  the  forward  edge  of 
the  battlefield  area.  It  includes  diagnosis,  isolation,  and  repair  of  equipment 
components.  The  following  figure  illustrates  the  TETS  operational  concept. 
(CCA  is  a  circuit  card  assembly) 
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The  following  is  a  chronological  list  of  significant  events  that  occurred  during 
the  TETS  program. 

TETS  Concept  Demonstration,  September  1992.  The  Marine  Corps 
Automated  Test  Support  Business  Center,  Albany,  Georgia,  developed  a 
prototype  system  constructed  from  commercial-off-the-shelf  equipment  for 
verification  and  proof  of  concept.  (The  prototype  was  referred  to  as  the 
automated  test  support  unit  prototype.)  The  prototype  system  successfully 
demonstrated  a  commercial  off-the-shelf  solution  capable  of  performing  all 
requirements  of  a  third  echelon  automatic  tester.  The  results  of  the  TETS 
concept  demonstration  prompted  the  Commander,  Marine  Corps  Systems 
Command,  to  establish  TETS  as  a  program. 

TETS  COEA,  January  1994.  The  Marine  Corps  conducted  a  COEA  to 
determine  the  most  cost-effective  solution  to  support  all  Marine  Corps  programs 
that  require  automatic  test  equipment.  Based  on  the  results  of  the  COEA, 
Marine  Corps  decisionmakers  determined  that  an  automatic  test  system  based  on 
the  automated  test  support  unit  prototype  provided  the  most  affordable, 
operationally  effective,  and  reliable  approach. 

Automatic  Test  System  Investment  Strategy,  January  1994.  In  response  to 
congressional  direction  to  establish  automatic  test  system  commonality,  the 
Office  of  the  Secretary  of  Defense  tasked  the  Institute  for  Defense  Analyses  to 
conduct  a  study  and  develop  an  investment  strategy  for  DoD  automatic  test 
systems.  The  study  concluded  that  implementing  an  investment  strategy  that 
uses  designated  DoD  standard  automatic  test  system  families,  instead  of  weapon 
system  unique  automatic  test  systems,  should  reduce  acquisition  cost.  The  report 
stated  that  CASS  and  IFTE  could  satisfy  95  percent  of  the  technical  test 
requirements. 

Automatic  Test  System  Policy,  April  1994.  The  Deputy  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  issued  a  policy  memorandum  on 
automatic  test  systems.  It  was  established  to  minimize  the  number  of  unique 
automatic  test  systems  being  introduced  into  DoD  field,  depot,  and 
manufacturing  operations,  and  to  encourage  use  of  commercial-off-the-shelf 
testers  and  components  thereby  reducing  the  logistics  burdens  and  long-term 
costs  to  DoD.  The  policy  required  DoD  Components  to  satisfy  all  acquisition 
needs  for  automatic  test  systems  by  either  using  designated  automatic  test 
system  families  or  obtaining  a  waiver.  The  Navy's  CASS  and  the  Army's  IFTE 
were  designated  as  the  initial  DoD  families.  Enclosure  4  discusses  additional 
issues  concerning  automatic  test  system  policy. 

TETS  Waiver  Request,  Fall  1994.  The  TETS  program  was  presented  to  the 
automatic  test  system  management  board  in  the  fall  of  1994,  for  waiver 
recommendation.  The  management  board  reviewed  the  prototype  and  requested 
that  concept  studies  be  conducted  on  the  CASS  and  IFTE  programs  to  determine 
the  feasibility  of  a  satisfactory  variant  to  satisfy  the  TETS  requirements. 
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Concept  Studies  for  CASS  and  IFTE,  March  1995.  Based  on  the  October  31, 
1994,  operational  requirements  document,  the  automatic  test  system 
management  board  asked  members  of  both  the  CASS  and  IFTE  programs  to 
study  the  feasibility  of  a  satisfactory  variant  to  satisfy  TETS  requirements.  The 
CASS  and  IFTE  concept  studies  concluded  that  neither  CASS  nor  IFTE  could 
satisfy  the  power,  reconfigurability  and  portability,  size,  and  weight 
requirements  of  TETS. 

Automatic  Test  System  Waiver,  May  1995.  The  automatic  test  system 
management  board  granted  a  waiver  for  TETS,  allowing  Marine  Corps  Systems 
Command  to  procure  an  automatic  test  system  that  can  satisfy  the  TETS 
requirements  through  open  bid  sample  competition.  The  management  board 
will  review  the  program  again  if  any  changes  are  made  to  the  current  acquisition 
strategy  or  schedule. 

Milestone  I/H,  May  1995.  A  combined  milestone  I,  concept  demonstration, 
approval  and  milestone  II,  development,  approval  for  TETS,  was  held  on 
May  11,  1995.  The  Milestone  Decision  Authority  considered  the  results  of  the 
TETS  COEA,  the  CASS  and  IFTE  concept  studies,  and  the  automatic  test 
system  management  boards  waiver  approval  among  the  various  other  required 
milestone  documentation  for  TETS.  Greater  emphasis  was  placed  on  the 
management  board's  recommendations  than  on  the  information  provided  by  the 
COEA.  As  a  result  of  the  milestone  I/II  review,  the  TETS  program  was 
authorized  to  enter  into  the  next  phase  of  the  acquisition  process,  specifically 
phase  II,  Engineering  and  Manufacturing  Development. 
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Automatic  Test  System  Policy.  Existing  automatic  test  system  standard 
families,  CASS  and  IFTE,  do  not  adequately  address  the  downsized  automatic 
test  system  requirement  like  the  TETS  program.  DoD  officials,  responsible  for 
program  oversight  on  automatic  test  system  policy,  expressed  concern  over  the 
potential  for  duplicative  investments  in  downsized  automatic  test  system  within 
and  across  weapon  programs  and  depots.  The  Marine  Corps  TETS,  the  Air 
Force  special  operating  forces  radio  frequency  mobile  electronic  test  set,  and  the 
Air  Force's  F-15  downsized  tester  are  examples  of  downsized  automatic  test 
system  applications.  To  address  those  concerns,  the  Office  of  the  Deputy  Under 
Secretary  of  Defense  for  Acquisition  and  Technology  clarified  existing  DoD 
policy  for  automatic  test  systems  that  will  be  published  in  January  1996.  This 
clarification  was  driven  by  significant  changes  in  DoD  acquisition  management 
processes,  specifically  die  Integrated  Product  and  Process  Development 
approach.  To  adequately  address  downsized  automatic  test  system 
requirements,  this  policy  should  include  sufficient  measures  to  ensure  that 
automatic  testers  are  acquired  on  a  standard  family  basis  and  duplication  is 
avoided.  An  interim  progress  report  on  the  proposed  implementation  of  the 
Integrated  Product  and  Process  Development  approach  is  tentatively  scheduled 
for  February  1996  and  a  final  report  is  expected  in  March  1996. 

Improving  the  COEA  Process.  The  Office  of  the  Assistant  Secretary  of 
Defense  for  Program  Analysis  and  Evaluation  has  recognized  the  need  to 
improve  die  COEA  process.  Personnel  from  Program  Analysis  and  Evaluation 
performed  a  review  to  examine  the  usefulness  of  COEAs  in  the  acquisition 
process,  the  level  of  effort  required  to  produce  a  COEA,  how  COEAs  should  be 
conducted,  who  should  perform  COEAs  and  the  timing  and  scope  of  COEAs. 
During  the  review,  Program  Analysis  and  Evaluation  personnel  contacted  the 
Institute  for  Defense  Analyses,  the  Center  for  Naval  Analyses,  and  staffs  and 
program  offices  of  the  Services.  They  completed  their  review  of  the  COEA 
process  in  July  1995.  Program  Analysis  and  Evaluation  personnel  made  initial 
recommendations  to  improve  the  COEA  process.  No  official  action  had  been 
taken  as  of  November  28,  1995. 
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Inspector  General,  DoD,  Reports 

Inspector  General,  DoD  (IG,  DoD),  Report  No,  95-024,  "Automatic  Test 
Systems  Acquisition,"  was  issued  on  November  4,  1994.  The  report  discusses 
efforts  to  achieve  commonality  in  standards  among  the  Military  Departments  as 
part  of  the  DoD  Policy  for  automatic  test  systems.  On  April  29,  1994,  the 
Deputy  Under  Secretary  of  Defense  for  Acquisition  and  Technology  issued  a 
DoD  policy  memorandum  to  improve  the  automatic  test  system  acquisition 
process.  The  intent  of  the  automatic  test  system  policy  is  to  minimize  unique 
types  of  automatic  test  systems  being  introduced  into  DoD  field,  depot,  and 
manufacturing  operations  and  to  encourage  the  use  of  commercial-off-the-shelf 
testers  and  components  throughout  DoD.  Execution  of  the  policy  will  reduce 
logistics  burdens  and  long-term  costs  to  DoD.  In  addition,  the  automatic  test 
system  Executive  Agent  Office  prepared  a  draft  directive  for  the  DoD  automatic 
•  test  system  acquisition  program  and  DoD  Instruction  5000.2  language  changes 
to  provide  guidance  for  the  implementation  of  automatic  test  system  policy. 
The  actions  taken  by  management  were  considered  responsive  and  the  report 
contained  no  recommendations.  As  of  December  1995,  the  draft  directive  and 
instruction  were  being  coordinated. 

The  IG,  DoD,  Report  No.  93-138,  "Quick-Reaction  Report  on  the  Acquisition 
of  the  F-15  Downsized  Tester,"  was  issued  on  June  30,  1993.  The  audit  was 
requested  because  the  Air  Force  was  not  developing  a  family  of  standardized 
automatic  test  equipment  or  using  existing  DoD  families  of  automatic  test 
equipment  that  met  multiple  weapon  system  and  Military  Department  test 
requirements.  The  report  states  that  the  Air  Force’s  development  and 
acquisition  of  unique  automatic  test  equipment  to  replace  existing  F-15  avionics 
automatic  test  equipment  was  not  cost-effective.  The  report  recommended  that 
the  Air  Force  discontinue  acquisition  of  unique  automatic  test  equipment  to 
replace  existing  F-15  avionics  automatic  test  equipment  and  prepare  thorough 
analyses  of  costs  and  technical  requirements  for  standardized  automatic  test 
equipment.  The  report  also  recommended  that  the  Under  Secretary  of  Defense 
for  Acquisition  and  Technology  and  the  Air  Force  implement  policies  and 
procedures  for  standardized  automatic  test  equipment.  The  Air  Force 
nonconcurred  with  the  recommendation  to  discontinue  acquisition  of  unique 
automatic  test  equipment  and  to  prepare  the  analyses.  The  Air  Force  stated  that 
discontinuing  the  acquisition  would  result  in  monetary  losses,  that  an 
operational  requirements  document  was  approved  by  the  Air  Force  Vice  Chief 
of  Staff,  and  that  a  COEA  was  not  required.  The  then  Assistant  Secretary  of 
Defense  (Production  and  Logistics)  and  the  Air  Force  agreed  to  implement 
policies  and  procedures  for  standardizing  automatic  test  equipment.  The  Under 
Secretary  of  Defense  for  Acquisition  and  Technology  directed  his  staff  to 
circulate  for  coordination  proposed  acquisition  policy  language  on  automatic  test 
systems.  In  the  interim,  his  staff  reviewed  automatic  test  system  acquisitions 
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against  the  objective  of  procuring  standardized  inventory  automatic  test 
equipment.  On  April  29,  1994,  the  Under  Secretary  of  Defense  for  Acquisition 
and  Technology  issued  a  DoD  policy  memorandum  for  automatic  test  systems. 

The  IG,  DoD,  Report  No.  92-095,  "Acquisition  and  Management  of 
Maintenance  and  Diagnostic  Automatic  Test  Equipment,"  was  issued  on 
May  21,  1992.  As  a  part  of  a  DoD-wide  audit  of  the  acquisition  and 
management  of  maintenance  and  diagnostic  automatic  test  equipment,  the  audit 
evaluated  the  effectiveness  of  DoD-wide  guidance  and  procedures  for 
monitoring  the  acquisition  and  management  of  maintenance  and  diagnostic 
automatic  test  equipment  by  the  Military  Departments.  The  report 
recommended  that  comprehensive  and  uniform  DoD-wide  policy  and  guidance 
on  the  acquisition  and  management  of  maintenance  and  diagnostic  automatic  test 
equipment  be  developed  and  implemented  and  that  the  Office  of  the  Secretary  of 
Defense  establish  oversight  responsibilities.  Management  agreed  on  a  need  for 
uniform  and  comprehensive  DoD-wide  policy  guidelines  on  the  acquisition  of 
automatic  test  equipment  and  the  need  to  clarify  Office  of  the  Secretary  of 
Defense  management  responsibilities.  An  automatic  test  system  study  was 
completed  and  an  investment  strategy  that  capitalizes  on  existing  automatic  test 
system  investments  was  developed.  The  investment  strategy  was  documented  in 
a  summary  report  that  recommended  management  changes,  acquisition  policy 
changes,  and  next  generation  technology  investments  to  verify  implementation 
and  long-term  viability  of  the  strategy.  On  April  29,  1994,  the  Under  Secretary 
of  Defense  for  Acquisition  and  Technology  issued  a  DoD  policy  memorandum 
for  automatic  test  systems. 

The  IG,  DoD,  Report  No.  92-037,  "Effectiveness  of  the  Air  Force's  Internal 
Controls  Over  the  Development  and  Acquisition  of  Maintenance  and  Diagnostic 
Systems,"  was  issued  on  January  23,  1992.  As  part  of  a  DoD-wide  audit  of  the 
development  and  acquisition  of  DoD  maintenance  and  diagnostic  systems,  the 
audit  evaluated  the  effectiveness  of  the  Air  Force's  principal  program  for 
monitoring  the  development  and  acquisition  of  maintenance  and  diagnostic 
systems,  the  modular  automatic  test  equipment  program.  The  report  states  that 
the  Air  Force  Systems  Command  product  divisions  and  the  Air  Force  Logistics 
Command  logistics  centers  were  not  complying  with  Air  Force  guidance  for 
acquiring  standardized  automatic  test  equipment.  As  a  result,  the  Air  Force 
experienced  a  continued  proliferation  of  equipment  and  provided  no  assurance 
of  acquiring  automatic  test  equipment  cost-effectively.  The  report 
recommended  that  the  Air  Force  develop  and  implement  an  effective  internal 
control  management  system  for  monitoring  the  development  and  acquisition  of 
automatic  test  equipment.  The  Air  Force  concurred  with  the  report  and 
accordingly  issued  Air  Force  Policy  Directive  63-2,  "Automatic  Test  Systems 
and  Equipment,"  July  19,  1994. 

The  IG,  DoD,  Report  No.  92-022,  "Development  and  Acquisition  of  DoD 
Maintenance  and  Diagnostic  Systems-Navy,"  was  issued  on  December  17, 
1991.  The  report  states  that  the  Navy's  plans  for  transitional  to  standard 
automatic  test  equipment  developed  under  its  CASS  Program  have  not  been 
fully  effective.  As  a  result,  potential  savings  opportunities  have  been  missed 
because  several  Navy  organizations  did  not  perform  work  load  and  economic 
analyses  to  determine  whether  it  was  feasible  and  economical  to  transition  from 
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existing  test  equipment  for  their  weapon  systems  to  CASS  test  equipment.  The 
report  recommended  that  the  Navy  develop  an  effective  internal  control 
management  system  to  monitor  the  Navy-wide  development,  acquisition,  and 
distribution  of  test,  measurement,  and  diagnostic  equipment.  In  response  to  the 
report,  the  Navy  nonconcurred  with  the  finding  and  all  recommendations. 
However,  as  a  result  of  mediation,  the  Navy  agreed  to  use  a  modified  Naval  Air 
Systems  Command  strategy  for  the  transition  to  automatic  test  equipment. 


Other  Prior  Reviews 

The  Institute  for  Defense  Analyses,  Paper  No.  P-2917,  "Investment  Strategy  for 
.  DoD  Automatic  Test  Systems,"  was  issued  in  January  1994.  In  response  to 
congressional  direction  to  establish  automatic  test  system  commonality,  the 
Office  of  the  Secretary  of  Defense  tasked  the  Institute  for  Defense  Analyses  to 
conduct  a  study  and  develop  an  investment  strategy  for  DoD  automatic  test 
systems.  The  study  states  that  implementing  an  investment  strategy  that  uses 
designated  DoD-standard  automatic  test  system  families,  instead  of  weapon 
system-unique  automatic  test  systems,  should  reduce  acquisition  cost.  The  report 
concluded  that  CASS  and  IFTE  could  satisfy  95  percent  of  DoD  automatic 
testing  requirements. 

The  Program  Manager,  Ground  Weapons  Marine  Corps  Systems  Command, 
issued  "Test,  Measurement  and  Diagnostic  Equipment  Study, "  on  December  22, 
1992.  The  study  was  commissioned  to  provide  information  on  the  available 
test,  measurement,  and  diagnostic  equipment  that  may  be  used  to  test  all 
systems  either  fielded  or  planned  for  which  the  Program  Manager,  Ground 
Weapons  is  responsible  for.  The  study  compares  the  costs  associated  with  the 
IFTE,  the  direct  support  test  set,  the  advanced  TOW  2  field  test  set,  and  the 
automated  test  equipment  program  proposed  by  the  Marine  Corps  Systems 
Command.  The  study  recommended  that  the  Program  Manager,  Ground 
Weapons  adopt  a  singular  suite  of  test  equipment,  the  direct  support  test  set. 
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Office  of  the  Secretary  of  Defense 

Assistant  Secretary  of  Defense,  Economic  Security,  Washington,  DC 

Assistant  Secretary  of  Defense,  Program  Analysis  and  Evaluation,  Washington,  DC 


Department  of  the  Army 

U.S.  Army  Test,  Measurement  and  Diagnostic  Equipment  Activity,  Huntsville,  AL 


Department  of  the  Navy 

Naval  Air  Systems  Command  Headquarters,  Director,  Automatic  Test  Systems 
Executive  Agent  Office,  Arlington,  VA 

Naval  Air  Systems  Command  Headquarters,  Consolidated  Automated  Support  System 
Program  Office,  Arlington,  VA 
Naval  Air  Warfare  Center,  Lakehurst,  NJ 
Marine  Corps  Combat  Development  Command,  Quantico,  VA 
Marine  Corps  Systems  Command,  Quantico,  VA 
Marine  Corps  Automatic  Test  Business  Center,  Albany,  GA 


Non-Government  Organizations 

GDE  Systems  Incorporated,  San  Diego,  CA 

GEC-Marconi  Avionics  Incorporated,  Atlanta,  GA 

Harris  Corporation,  Melbourne,  FL 

Hewlett  Packard  Company,  Rockville,  MD 

Institute  for  Defense  Analyses,  Arlington,  VA 

JB  Systems  Engineering  Support  Company,  Huntsville,  AL 

Lockheed  Martin,  Orlando,  FL 

National  Instruments,  Austin,  TX 

Pentastar  Electronics  Incorporated,  Huntsville,  AL 

Racal  Instruments,  Irvine,  CA 

Tektronix  Incorporated,  Atlanta,  GA 
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Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Director,  Defense  Logistics  Studies  Information  Exchange 
Deputy  Assistant  Secretary  of  Defense,  Industrial  Affairs 
Under  Secretary  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Deputy  Under  Secretary  of  Defense  (Logistics) 

Assistant  to  the  Secretary  of  Defense  (Public  Affairs) 


Department  of  the  Army 

Auditor  General,  Department  of  the  Army 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Navy 
Commandant  of  the  Marine  Corps 

Commander,  Marine  Corps  Combat  Development  Command 
Commander,  Marine  Corps  Systems  Command 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Air  Force 


Other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Logistics  Agency 
Director,  National  Security  Agency 

Inspector  General,  National  Security  Agency 


Non-Defense  Federal  Organizations  and  Individuals 

Office  of  Management  and  Budget 

National  Security  and  International  Affairs  Division,  General  Accounting  Office 
Technical  Information  Center 
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Non-Defense  Federal  Organizations  and  Individuals  (cont'd) 

Defense  and  National  Aeronautics  and  Space  Administration  Issues 
Military  Operations  and  Capabilities  Issues 
Chairman  and  ranking  minority  member  of  each  of  the  following  congressional 
committees  and  subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 
House  Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal 
Justice,  Committee  on  Government  Reform  and  Oversight 
House  Committee  on  National  Security 
Honorable  Howell  Heflin,  U.S.  Senate 
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Audit  Team  Members 

This  report  was  prepared  by  the  Logistics  Support  Directorate,  Office  of  the 
Assistant  Inspector  General  for  Auditing,  DoD. 

Shelton  R.  Young 
John  A.  Gannon 
Stephen  T.  Hampton 
Timothy  J.  Harris 
Chandra  P.  Sankhla 
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